esp@cenet - Document Bibliography and Abstract 



Page 1 of 1 



Method for maintaining a sequence of events function during failover in 
a redundant multiple layer system 



Patent Number: 

Publication date: 
Inventor(s): 
Applicant(s): 
Requested Patent: 
Application 
Priority Number(s): 
IPC Classification: 
EC Classification: 
Equivalents: 



r US5426774 
1995-06-20 

BANERJEE INDRA (US); MCLAUGHLIN PAUL F (US); MCCRACKEN KEVIN R 

HONEYWELL INC (US) 

DE69414392T 

US1 9930042923 19930406 

US 1 9930042923 1 9930406 

G06F11/00 

G06F11/14A4C 

DE69414392D, T EP0707722 (WQ9423367), B1, l~ W09423367 



Abstract 



In a process control system having a redundant multilayer hierarchical structure, each node of a layer being 
redundant sequence of events inputs are received from field devices by an input/output processor (IOP) 
The IOP is a digital input sequence of events (DISOE) IOP, the IOP being the lowest layer of the hierarchical 
structure. The IOP interfaces with a controller at the next layer of the hierarchy. A method for reliably 
maintaining a sequence of events function during a failover of any of the redundant nodes comprising the 
steps of maintaining a log, a circular list, by the local DISOE. The circular list is a rolling log of all sequence 
of events data for a predefined time period. When a failover occurs, the new primary commands an event 
recovery. The event recovery process freezes the log and uses the information in the log to recreate the 
events data. The freeze operation inhibits background purge activity for the log thereby avoiding the deletion 
of information past the defined time. New events data is still entered in the log. Once the log has been 
processed the freeze operation is negated. The recreated data is transmitted to the controller in accordance 
with a predefined protocol, thereby avoiding the loss of any events data as a result of the failover 
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Des rlption 

BACKGROUND OF THE INVENTION 

The present invention relates to a method of fai lover 
in redundant systems, and more particularly, to a meth- 
od for maintaining a sequence of events function during 
a failover in a redundant system having multiple layers. 

In a process control system, the system of the pre- 
ferred embodiment, intermediate layers (of a hierarchi- 
cal scheme of the system) between an input device and 
a historization device have redundancy due to process 
control security needs. As a result, the sequence of 
events information could be lost in the event of a failover 
Thus, there is provided by the present invention a meth- 
od which maintains the sequence of events information 
from multiple nodes of a network even if one or more of 
these nodes should sustain a failover. 

European Patent Application Publication No. EP 
0434532 describes a method of inputting data into a 
large memory store. 

The method and system described in this document 
are similar as in the present application, and show fun- 
damentally the features as in the appended Claim 1 ex- 
cept for the details in features b) and c). 

SUMMARY OF THE INVENTION 

The present invention provides a method for main- 
taining a sequence of events function during a failover 
of any redundant nodes in a process control system hav- 
ing a redundant multilayer multimode hierarchical struc- 
ture, each node of a layer being redundant, wherein se- 
quence of events inputs are received from field devices 
by a receiving input/output processor (IOP) being a dig- 
ital input sequence of events (DISOE) IOP, the IOP be- 
ing the lowest layer of the hierarchical structure, the IOP 
interfacing with a controller 30 at the next layer of the 
hierarchy, the method comprising the steps of;- 

a) receiving events data from said field devices by 
the receiving IOP 

b) maintaining a first buffer with events data, said 
first buffer being a circular list, said events data be- 
ing time stamped by the receiving IOP with a rela- 
tive time of receipt and a sync message ID, the rel- 
ative time of receipt being a number of clock times 
since receipt of a time synchronizing message from 
the controller, the time synchronizing message fur- 
ther including the sync message ID, said events da- 
ta in the first buffer being the events data received 
since a previous transmission of said events data 
to the controller, 

c) maintaining a second buffer with events data, 
said second buffer being a circular list, said events 
data being time stamped by the receiving IOP with 
the r iative time of receipt, said events data in the 
second buffer being events data rec ived within a 



predetermined period such that a history of events 
is maintained for the predetermined period; 
d) upon request from the controller, transmitting the 
events data and the corresponding relative time of 
s receipt from the first buffer to the controller whereby 
the controller calculates a real time of occurrence 
of the events data, the real time of occurrence being 
time synchronized to the process control system; 
and 

10 e) when a failover is detected, upon a request for 
event recovery from any of the hierarchical layers, 
recreating the events data from the second buffer 
to transmit the recreated events data to the control- 
ler in accordance with a predefined protocol thereby 

is avoiding the loss of any events data as a result of 
the failover. 

Preferably, the step of maintaining a second buffer 
comprises the steps: 

20 

a) during a failover, freezing a purge operation of 
the events data in the second buffer even though 
the events data is older than the predetermined pe- 
riod of time; 

25 b) continuing to store events data in the second 
buffer during the failover thereby not losing any 
events data; and 

c) upon completing processing events data in the 
second buffer, unfreezing the purge operation of the 
30 second buffer to eliminate out of date events data. 

The present invention also provides a process con- 
trol system having a redundant multilayer multimode hi- 
erarchical structure. 

35 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figure t shows a block diagram of a process control 
system in which the present invention can be uti- 

40 lized; 

Figure 2 shows a block diagram of a process con- 
troller, including I/O modules (IOP), in which the 
present invention can be utilized; 
Figure 3 shows a block diagram of a controller 

45 which is included in the process controller of Figure 
2; 

Figure 4 shows a block diagram of an I/O module 
which is included in the process controller of Figure 

2; 

so Figure 5 shows a simplified block diagram of the 
process controller of Figure 2; 
Figure 6 shows a block diagram of the preferred em- 
bodiment of the process control system which in- 
cludes a message recognizer for achieving time 

55 synchronization of the hierarchical layers of the 
process control system; 

Figure 7 shows a block diagram of a message rec- 
ognizer of the preferred embodiment of the process 
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control system; 

Figure 8 shows a time line example of events which 
occur in the synchronizing of two networks; and 
Figure 9 shows a block diagram of the process con- 
trol system which includes the functions necessary 
to support the method of the present invention. 

DETAILED DESCRIPTION 

Before describing the method of the present inven- 
tion, it will be helpful in understanding a system environ- 
ment in which the present invention can be utilized. Re- 
ferring to Figures 1 and 2, there is shown a block dia- 
gram of a process control system 10 in which the 
present invention can be found. 

The individual layers of the process control systems 
include an input processor device (IOP 21) used to de- 
tect digital state change activity, a controller 30, 40, (in- 
put processor devices 21 and the controller 30, 40 make 
up a process controller 20), a network interface (NIM 
602) which allows a network of process controllers 20 
to be interfaced to a local area network (plant control 
network 11) which includes man machine interface de- 
vices (US 122) and a historization device (HM 126). The 
present invention allows for any of these interposing de- 
vices to be made redundant for the critical needs of proc- 
ess control, while ensuring that SOE information is not 
lost due to the failover of any one or more nodes in the 
hierarchy. The essential method by which this is accom- 
plished is to conjoin the existing function of event recov- 
ery with a new mechanism whereby the Input Processor 
can recreate SOE information following the failover of 
an upper-layer node or a failover from a redundant part- 
ner, as will be described hereinunder. 

The structure of the process control system 10 will 
now be described. Referring to Figure 1 , there is shown 
a block diagram of the process control system of the pre- 
ferred embodiment. The process control system 10 in- 
cludes a plant control network 1 1 , in which a process 
controller 20 is operatively connected to the plant control 
network 11 via a universal control network (UCN) 14 to 
a network interface module (NIM) 602. In the preferred 
embodiment of the process control system 1 0, addition- 
al process controllers 20 can be operatively connected 
to the plant control network 11 via the same UCN 14, 
and additional UCNs 14 can be added to the plant con- 
trol network 11 via additional corresponding NIMs 602. 
The process controller 20, interfaces analog input and 
output signals, and digital input and output signals (A/I, 
A/O, D/l, and D/O, respectively) to the process control 
system 10 from the variety of field devices (not shown) 
which include, pumps, motors, valves, pressure switch- 
es, pressure gauges, thermocouples,.... Inputs also in- 
clude relay closures and the like indicative of the occur- 
rence of a predefined event. 

The plant control network 11 provides the overall 
supervision of a controlled proc ss, in conjunction with 
the plant operator, and obtains all the information need- 



ed to perform the supervisory function, and includes an 
interface with the operator. The plant control network 1 1 
includes a plurality of physical modules, which include 
a universal operator station (US) 122, an application 

s module (AM) 124, a history module (HM) 126, a com- 
puter module (CM) 128, and duplicates of these mod- 
ules (and additional types of modules, not shown) as 
necessary to perform the required control/supervisory 
function of the process being controlled. Each of these 

10 physical modules is operatively connected to a local 
control network (LCN) 120 which permits each of these 
modules to communicate with each other as necessary. 
The NIM 602 provides an interface between the LCN 
120 and the UCN 14. A more complete description of 

is the plant control network 11, and the physical modules 
can be had by reference to U.S. Patent No. 4,607,256. 

Referring to Figure 2 there is shown a block diagram 
of the process controller 20. The preferred embodiment 
of the process controller 20 of the preferred embodiment 

20 of the process control system 1 0 includes a controller A 
30 and a controller B 40, which effectively operate as a 
primary and secondary controller. Controller A 30 and 
controller B 40 are connected to the UCN 14, the UCN 
1 4 in the preferred embodiment, comprising for commu- 

25 nication redundancy purposes, a UCN(A) 14A and a 
UCN(B) 14B. Input/output processors (lOPs) (some- 
times referred to herein as input output (I/O) modules) 
21 interface to field devices, field devices being various 
pumps, motors, valves, pressure switches, pressure 

30 gauges, thermocouples,... which can be analog inputs 
(A/I), analog outputs (A/O), digital inputs (D/l), and dig- 
ital outputs (D/O). The controller A 30 and controller B 
40 interface to one or more I/O modules via a bus 22, 
the bus 22 in the preferred embodiment, comprising for 

3S communication redundancy purposes, a bus 22A and a 
bus 22B. 

Controller A and controller B, 30, 40, can commu- 
nicate with each other via three mediums, the UCN 14, 
a link 1 3 between the controllers, and the buses 22A, 

40 22B, with bus 22A and bus 22B in the preferred embod- 
iment being serial I/O links. One controller (controller A 
30 or controller B 40) operates as a primary controller 
and the other controller operates as a secondary con- 
troller (in more of a reserve mode than a back-up, in that 

45 if a failure of controller A 30 should occur, controller B 
is ready to take over the control function with essentially 
no start-up or initialization time). On a predetermined 
time basis, point processing is performed by the control- 
ler designated as the primary controller and communi- 

so cates with the I/O modules 21 . In addition, the controller 
acting as the primary controller communicates with the 
plant control network 11 reporting status, history, and 
accepting inputs from the plant control network such as 
commands from the operator via the universal station 

55 122. In addition, a data base maintained by the primary 
controller is communicated to the secondary controller 
via link 1 3. As mentioned above, one controller operates 
as a secondary controller; however, it will be understood 
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by those skilled in the art that a secondary control! r is 
not necessary for the proc ss controller 20. A more 
complet description of process controller 20, which in- 
cludes controll r 30 and IOP 21, can be had by r fer- 
ence to U.S. Patent No. 5,146,401. 

Referring to Figure 3, there is shown a block dia- 
gram of the controller 30, 40. A modem 50 is connected 
to the UCN 14, the modem having two inputs, one con- 
nected to UCN 1 4A and the other connected UCN 1 4B. 
The modem 50 interfaces with a communication unit 
(COMM) 60 which in turn interfaces with a global mem- 
ory 70, an I/O interface unit 80, and a control unit 90 via 
global bus 72. The communication unit 60 includes a 
communication control unit, in the preferred embodi- 
ment a token bus controller (TBC) 61, Motorola type 
68824, which is connected to a local bus 62. A processor 
A 63 (which essentially performs the communication 
function) is connected to the local bus 62, and a local 
memory A 64, which is also connected to the local bus 
62. The processor A 63 communicates with the plant 
control network 1 1 via modem 50 and TBC 61 . The local 
memory A 64 stores information, including personality 
image which is downloaded from the plant control net- 
work 11, for use by processor A 63 and TBC 61. The 
global memory 70 stores information which is common 
to both processor A 63 and a processor B 91. It also 
stores all the data received from bus 22A and bus 22B. 
The global memory 70 also serves as an interprocessor 
communication vehicle between the processors A 63 
and B 91. Control unit 90 includes the processor B 91 
and a local memory B 92, both connected to a local bus 
93. Processor B 91 performs the control function (i.e., 
control processing) relating to the field devices. This es- 
sentially includes performing the point processing, and 
updating the local memory B 92 and global memory 70. 
Also coupled to the local bus 93 of control unit 90 is a 
track unit (not shown) which is utilized to implement the 
data base transfer via link 1 3 to the other controller 30, 
40 of the process controller 20. 

The I/O interface unit 80 includes a receiver-trans- 
mitter device, this device being a UART (Universal 
Asynchronous Receiver/Transmitter) 81 . The UART 81 
is coupled through drivers 82, 83 to bus 22A and bus 
22B, respectively. 

Processor B 91 receives data from the various field 
devices through global memory 70, performs the nec- 
essary point processing and control function, and then 
updates the local memory B 92 and global memory 70, 
as required. The communication unit 60, in response to 
commands from the control unit 90 via global memory 
70, inputs and outputs data between the I/O modules 
21 (via the I/O interface unit 80) and the global memory 
70, thereby relieving the control unit 90 from the burden 
of I/O module management. In this manner the control 
processing is performed by the control unit 90 within the 
process controller 20 for the predefined attached field 
devices, and the communication (i.e., the I/O control) is 
handled by the communication unit 60 through the 



UART 81. 

Referring to Figur 4 th r is shown a partial block 
diagram of an I/O module of th components of interest. 
A transceiver (anti -jabber circuit) 201 interfaces with bus 
s 22A and bus 22B. The transceiver 201 interfaces with 
a microcontroller (u-controller) 202 which, in the pre- 
ferred embodiment, is of the type, Intel 80C31 . The mi- 
crocontroller is coupled to a local bus 203, and includes 
an EPROM 204 and a RAM 205 also connected to the 
local bus 203. The RAM 205 contains the information 
which forms the database for the I/O module 21 . The 
EPROM 204 contains the program information utilized 
by the microcontroller 202. The application specific cir- 
cuits 209 are also connected to the local bus 203, and 
the microcontroller 202 via the local bus 203. The appli- 
cation specific circuits 209 vary from I/O module to I/O 
module depending on the field device to which the I/O 
module is to be coupled. If the field device is of a type 
which requires a digital input, then the application spe- 
cific circuit 209 will include the logic in order to place the 
digital input into a predefined format which will interface 
with the remainder of the I/O module. Likewise, if the 
field device is such that requires an analog input, then 
the application specific circuit contains logic which con- 
verts the analog input signal (via an A/D converter) into 
a format again consistent with predefined formats. In 
this manner, the I/O modules are referred to as a specific 
I/O module type. The microcontroller 202 performs the 
I/O processing (or preprocessing) for the application 
specific circuits 209. The preprocessing will vary from 
each I/O module 21 depending on the type (i.e., A/I, A/ 
O,...) the preprocessing essentially consisting of trans- 
lating the signals from the application specific circuits to 
a format compatible with the controller 30, 40, and 
putting the signals from controller 30, 40 in a format 
compatible with the I/O module 21 . Some of the preproc- 
essing performed includes zero drift, linearization (line- 
arizing thermo-couples), hardware correction, compen- 
sation (gain compensation and zero compensation), ref- 
erence junction compensation, calibration correction, 
conversions, checking for alarms (limits)... and gener- 
ating a signal in a predetermined format having prede- 
termined scale (i.e., engineering units, normalized units, 
percent of scale,...). In the preferred embodiment seven 
types of applications specific circuits are provided for, 
these include a high level analog input, low level analog 
input, analog output, digital input, digital output, smart 
transmitter interface, and pulse input counter. 

The IOP redundancy will now be described. Refer- 
ring to Figure 5, there is shown a simplified block dia- 
gram of the process controller 20 of Figure 2, having the 
redundancy of the controller omitted, and having an IOP 
and a backup IOP, only, for purposes of example. In the 
preferred embodiment, up to forty (40) lOPs can be in- 
cluded, and any mix of IOP types can be included in a 
redundant or non -redundant configuration. Note option- 
al redundancy of each IOP. As will be recognized by 
those skilled in the art from the description above, the 
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controller 30 performs as the master processor, the IOP 
module 21 -A as th primary slave proc ssor, and th 
IOP module 21 -B as the backup (or secondary or redun- 
dant) slave processor. 

Initially, the controller 30 acts to synchronize the 
IOP(B) 21 -B (assuming, of course, that IOP(B) is 
present). Synchronizing is the process whereby the 
same data base is contained in both IOP(A) 21 -A and 
IOP(B) 21 -B. The information of the data base of IOP 

(A) is requested by the controller 30. IOP(B) 21 -B eaves- 
drops on the transmissions of data from IOP(A) 21 -A to 
the controller 30 and stores the information in its data 
base memory thereby causing the data base of IOP(B) 
21 -B to be the same, whereupon IOP(B) is commanded 
to start executing. IOP(B) performs the same operations 
as IOP(A). At essentially the same time (however, each 
IOP is operating using its own clock). It will be recog- 
nized that IOP(B) 21 -B is a dedicated backup. Once IOP 

(B) is synchronized, the controller data base is updated 
to indicate the status of IOP(B). Further details describ- 
ing the communication scheme between the controller 
30 and the IOP(A) and IOP(B), and the method of 
failover from IOP(A) to IOP(B) can be had by reference 
to U.S. Patent No. 5,1 36,498, assigned to the same as- 
signee as the assignee of the present application. 

Generally, hierarchical networks are software-driv- 
en and have variable/unpredictable delays in the net- 
work hardware and software between the time messag- 
es are queued for transmission and actually transmitted, 
and between the time messages are received and ac- 
tually processed. This lack of determinism prevents syn- 
chronization or time broadcast via messages containing 
high-precision clock times. The method of time synchro- 
nization of the preferred embodiment will now be de- 
scribed. Referring to figure 6, there is shown a block di- 
agram of the preferred embodiment of the process con- 
trol system 10 which includes a time synchronization 
scheme between the various hierarchical layers. The 
NIM 602 of the plant control network 1 1 includes a mes- 
sage recognizer-X 1 30, and controller A 30 and the re- 
dundant controller B 40 if applicable, of the process con- 
troller 20, each include a message recognizer-Y 95. 

Referring to Figure 7, there is shown a block dia- 
gram of the message recognizer 130, 95. An oscillator 
301 drives a free running counter (FRC) 305. A first and 
second latch 310, 311 is included and permits the con- 
tents of the FRC 305 to be stored in the latches upon 
command. A message detector 315 samples the mes- 
sages being transmitted or received. The message de- 
tector 315 provides the first control signal, CONT 1, 
which latches the contents of the free running counter 
305 into the first latch 310 when a unique header of a 
synchronization message is transmitted or received. 
This latching is essentially simultaneous in all devices 
on the same network bus and provides the basis for all 
subsequent synchronization activities. No other types of 
messages latch the free running counter 305. The sec- 
ond latch 31 1 of the message recognizer 1 30, 95 is used 



in various ways to provide the accurate synchronization 
times required for various applications of th pres nt in- 
vention. In the preferred embodiment of th pres nt in- 
vention a s cond control signal, CONT 2, is provided to 

5 cause the second latch 311 to latch upon a predeter- 
mined software command. The control of the second 
latch will be explained in further detail hereinunder. The 
first and second latches 310, 311, are both readable by 
a corresponding computer included in the network. The 

10 length and resolution of the free running counter 305 
and the latches are determined by the time resolution 
desired, the synchronization interval, and other factors 
well known by those skilled in the art. Although the pre- 
ferred embodiment of the present invention utilizes the 

is IEEE 802.4 communication scheme, other communica- 
tion protocols can be used, including IEEE 802.3, as will 
be recognized by one skilled in the art. 

The method of time synchronization between hier- 
archical layers will now be described. The time (i.e., real 

20 time in digital format in terms of hours, minutes, sec- 
onds) within the plant control network 11 is referred to 
as "LCNTIME". The time is generated by timing subsys- 
tems of each of the physical modules 1 22, 1 24, 1 26, and 
the NIM 602 (the NIM being a node coupled to the LCN 

25 1 20). The timing subsystem of each of the physical mod- 
ules are denoted by the letter X. The clock-X maintained 
by each of the physical modules are all the same, the 
clocks being synchronized under control of a bus con- 
troller 128, in a manner well understood by those skilled 

30 in the art; specifically, the manner is described in U.S. 
Patent 4,890,222, "Apparatus for substantially synchro- 
nizing the timing subsystem of the physical modules of 
a local area network", the patent being assigned to the 
same assignee of the present application. 

35 The controllers 30, 40 each have their respective 
timing subsystems, denoted as Y, which has a controller 
time, CONTIME, which can be skewed from the LCN- 
TIME. 

Referring to Figure 8 there is shown a time line ex- 

40 ample of the events which occur in the synchronizing of 
the two networks, CONTIME being synchronized to LC- 
NTIME. In the preferred embodiment every six seconds 
a synchronization message is transmitted by the NIM 
602. The synchronization message has a format which 

45 is detectable by the message recognizer 1 30, 95 and 
further contains a message number. The message 
number is a sequential number which identifies the 
number of messages which have been transmitted. 
When the synchronization message is transmitted over 

50 the UCN 14, at time t n , the message recognizer-X 130 
and the message recognizer-Y 95 each detect that a 
synchronization message was transmitted/received, re- 
spectively. Thus, each first latch 310 latches the con- 
tents of the respective free running counter 305. As- 

ss sume for purposes of example that the free running 
counter (FRCX) 305 of the message recognizer-X (MR- 
X) 1 30 has a value of 6 at the time the synchronization 
message is actually transmitted over the UCN 14, the 
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synchronization message having a message number of 
78. Sine the actual transmission time ov r the UCN 1 4 
is essentially zero, the first latch (LAT1X) of MR-X 130 
latch s the contents of FRCX such that the cont nts of 
LAT1X is also a 6. Further, the first latch (LAT1 Y) 310 
of the message recognizer (MR-Y) 95 latches the con- 
tents of the free running counter (FRCY) 305 of the MR- 
Y 95, such that in this example the contents of FRCY 
and LAT1 Y are both 107, the time of the actual trans- 
mission of the synchronization message over the UCN 
14 occurring at time t n . 

Assume in this example at t n + 1 nothing occurs. 
Again in this example at t n + 2 , the control software of the 
IMIM 602 recognizes that a synchronization message 
was transmitted. The control software of the NIM 602 
reads the current value of the clock-X, i.e., LCNTIME, 
and causes the second control signal CONT2 of the MR- 
X 1 30 to be generated such that the contents of the sec- 
ond latch (LAT2X) of the MR-X 130 contains the value 
of 8, the current value of FRCX 305. The NIM 602 then 
calculates the real time the sync message was transmit- 
ted in accordance with the following formula: 

TIMESYNC = CLOCK X - (LAT2X-LAT1X). 

This time, denoted as LCNTIME1 , is the real time that 
the message having serial number 78 was transmitted. 
The NIM 602 sends a second message to the controller 
30 which includes the value of TIMESYNC and the mes- 
sage number in this example the message number was 
78. This message has a format such that it is not recog- 
nized by either message recognizer 130, 95. Upon re- 
ceipt of this second message the controller associates 
the contents of LAT1 Y with LCNTIME1 thereby synchro- 
nizing the controller to the LCNTIME. Thus the preferred 
embodiment of scheme the time synchronization is a 
two message pair. Thus, for the controller 30 in this ex- 
ample, the value in LAT1 Y corresponds to LCNTIME1 . 
Someone skilled in the art would recognize that the two 
message pair could be implemented in one message 
type by including the real time of the previous sync in 
the data field of the next sync message. 

Asynchronously, every two second in the preferred 
embodiment of the time sync scheme, a synchronization 
message is transmitted by the controller 30 over the bus 
(I/O link) 22 to the IOP 21 . At some time, LCNTIME 2, 
the sync message is sent over the I/O link 22. At the 
time that the sync message is transmitted by the I/O link 
22 the second latch (LAT2Y) 311 of MR-Y 95 is caused 
to latch the contents of the FRCY 305 by the I/O inter- 
face 80. In the example, the FRCY 305 has a value of 
128 and the contents stored in LAT2Y is also 128 as a 
result of the latching process. The sync message to the 
IOP 21 also contains a message number (the message 
number is 24 in this example), the message number to 
the IOP being indep ndent of the messag numbers re- 
ceived from the NIM 602. The controller 30 calculates 



LCNTIME2, the time the sync message was sent to the 
IOP as follows: 

LCNTIME2 = LCNTIME1 + (LAT2Y - LAT1 Y). 

The controller maintains a list of sync ID numbers 
and the corresponding LCN times for the last 300 min- 
utes worth of sync ID values, in the preferred embodi- 
ment. This table is also maintained in the backup con- 
troller. After each IOL time sync broadcast, the primary 
controller transfers the IOL sync ID number and the cor- 
responding LCN time to the backup controller over the 
UCN and the IOL. 

The IOL time sync message resets a time counter 
maintained in the IOP; thus, LCNTIME2 corresponds to 
zero time in the IOP 21 . Thus a time base is established 
for the I/O link 22. When the IOP 21 detects an event 
change on one of the inputs from the field devices, the 
occurrence of the event is associated with a value in the 
time counter maintained in the IOP. At some time when 
the controller requests data from the IOP (recall in the 
preferred embodiment of the present system that the 
IOP 21 only responds to request from the controller), the 
IOP transmits to the controller the input point of the 
event, the value of the time counter when the digital 
event occurred, plus the message number of the sync 
ID message. Thus in this example the message number 
will have a message ID of 24. the sync time will have 
the value of the time counter maintained in the IOP when 
the event occurred, and the point number will corre- 
spond to the input point (if the IOP has 32 inputs, the 
point number will have a value of between 1 to 32 de- 
pending on the input point of the event. The input point 
is associated with the function of the event which was 
read. When the controller receives the information the 
controller adds the value of the time counter maintained 
in the IOP to the LCNTIME2 to obtain the actual time of 
the occurrence of the event. 

The preferred embodiment of the present invention 
describes a method and apparatus for providing time 
synchronization of data transmitted between devices of 
bus type local area networks. In addition an alternative 
method is described in which the real time is maintained 
by a master device (the controller) rather than in the 
slave device (IOP). It will be obvious to one skilled in the 
art that the timed synchronization process between the 
controller and the IOP can be implemented in the same 
fashion as the synchronization between the LCN net- 
work and the controller by including the message rec- 
ognizer apparatus internal to the lOP's. 

Referring to Figure 9, there is shown a block dia- 
gram of the process control system which includes the 
elements necessary to support the method of the 
present invention. There is included a digital input IOP 
21 A, B (referring to herein as DISOE) to which digital 
information indicative of vents data is inputted. The dig- 
ital events data, sequence of events (SOE) data, is time 
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tagged by the DISOE. When the state chang of any 
input changes, that information is stor d and time 
tagged by the DISOE with th r lative time and sync ID 
number is accordance with the scheme described 
above. The IOP 21 (DISOE), which is the element which 
detected the occurrence of the event, is the element 
which time tags the data. In previous systems, the time 
stamp (or tag) was performed by some other node at a 
different layer in the hierarchy; namely the NIM 602, 
which is at time later the actual time of occurrence of the 
event. Therefore, the time resolution of the method of 
the preferred embodiment is significantly enhanced. 

As described above one controller, controller A 30, 
is operating as a primary controller and controller B 40 
is operating as a secondary. Similarly IOP-A 21 A is op- 
erating as a primary IOP and IOP-B 21 B is operating as 
a secondary IOP The inputs from the field devices go 
to both lOPs 21 A, 21 B, via a terminal board 251. The 
communication between the controller 30 and the IOP- 
A 21 A is as described above. Recall that IOP-B 21 B is 
executing the same task as the primary IOP 21 A, and 
is eavesdropping on all the communications between 
the controller A30 and IOP-A 21 A. Time sync is a broad- 
cast over the I/O link and is received equally by all lOPs 
present on the lOL. When an event occurs, the relative 
internal clock of each IOP plus the sync ID is tagged 
with the input data in each IOP 21 A, 21 B. The events/ 
time data is stored in each IOP. The controller A30 pe- 
riodically requests the events/time data from IOP-A, cal- 
culates the real time, and subsequently transfers that 
information to the process control network, including the 
history module 126. 

Each DISOE IOP 21 includes two buffers a PV cir- 
cular list (or more simply circular list or list) and a PV 
change log (or more simply change log or log). The cir- 
cular list contains all the events (with time of occurrence) 
created by the IOP based on the change status of the 
digital inputs for collection by the controller 30. When 
the events from the list are collected by the controller 
30, that data in the list is essentially gone. However, the 
change log maintains the events data (and correspond- 
ingtime tag) in the change log buffer for a predetermined 
period of time. In the preferred embodiment, data in the 
log is kept for the past 20 seconds, this time being long 
enough to ensure that events are saved for recovery due 
tofailover at any layer in the hierarchy. Recall that in the 
preferred embodiment, requests from the controller for 
events data from the IOP-A is every half second. There- 
fore, in the event a malfunction and failover occur, any 
events data is not lost. The maintenance of the log and 
list is via pointers well known to those skilled in the art. 

There exists in the IOP sufficient information to rec- 
reate the events following a system alarm recovery com- 
mand from a node in the hierarchy. Event recovery pro- 
vides the capability whereby alarms and other dynamic 
event data is r cr ated in r sponse to the recovery com- 
mand. It is the proc ss in which ev nts ar recov red 
following a node failover. Therefore, if a node such as 



the NIM 602 or the controller 30 undergoes a failover; 
the new primary commands an event recovery to ensure 
that no event data is lost. The operations associated 
with the log are two-fold. First, a continuous background 
s operation processes the log and discards any SOE data 
that are older than the defined time period. Secondly, 
the event recovery process has the capability to 'freeze' 
the log and use the log's information to recreate SOE 
data. This freeze operation means that while frozen, the 
io background purge activity is blocked from deleting in- 
formation past the defined time. It is not frozen with re- 
spect to new SOE information; SOE data that occurs 
after event recovery starts is placed in the log so that it 
is not lost. Once the event recovery operation has proc- 
is essed the log, it unfreezes the list and the background 
operation can eliminate out-of-date information. The 
freeze permits a node which fails over to command re- 
covery within 20 seconds, in the preferred embodiment. 
Some total system operations will now be described 
20 in order to give an overview of some failover scenario's. 
First consider the case of the failover of a SOE Input 
Processor. Redundant Input Processors are synchro- 
nized, such that they have the exact same configuration 
data base. Both partners are connected to the physical 
25 inputs, and are processing digital state changes and 
producing SOE time ordered information. Upon failure 
of the primary, the secondary fulfills the role of primary 
and is commanded to recreate all event information, in- 
cluding SOE data. It does so using the change log; the 
30 change log has all SOE information for the last n sec- 
onds (n being the system design constant). This infor- 
mation is then made available to the Controller via the 
appropriate event reporting mechanism. The Controller 
then has the responsibility to report this data to its next 
35 layer, such that data will eventually reach the historiza- 
tion device as if it has originated in the original primary 
Input Processor. It is possible to receive the same infor- 
mation more than once, but duplicates are acceptable 
whereas lost information is not. 
40 The next case is a failover of the Controller. The im- 
portant item to consider is that when a Controller fails 
over to a Secondary, the Secondary will command all I/ 
O processors to recreate all event information and then 
will pass this data to the upper layers. The SOE Input 
45 Processor will do so using the same method as is done 
for its own failover. Note that redundant SOE Input Proc- 
essors are not required in order to have Controller re- 
dundancy. 

Finally, consider the case of NIM 602 redundancy. 

50 On the occurrence of its failover, the NIM secondary de- 
vice will command all Controllers to recreate all events 
within their domain. Hence, this is an extrapolation of 
Controller redundancy, and is satisfied using an identi- 
cal method. Further details regarding the Controller and 

55 NIM failover, and the IOP failover operations can be had 
by referenc to U.S. Patent No. 5,01 6,244 and U.S. Pat- 
ent No.5,1 36,498, respectively, both patents being as- 
signed to the same assigne as the present applica- 
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tions. 

While there has been shown what is considered th 
preferred embodiment of the present invention, it will b 
manifest that many changes and modifications can be 
made therein without departing from the scope of the 
invention as defined in the annexed claims. 



Claims 



A method for maintaining a sequence of events 
function during a failover of any redundant nodes in 
a process control system (10) having a redundant 
multilayer multimode hierarchical structure, each 
node of a layer being redundant, wherein sequence 
of events inputs are received from field devices by 
a receiving input/output processor (IOP) (21 A, 21 B) 
being a digital input sequence of events (DISOE) 
IOP, the IOP being the lowest layer of the hierarchi- 
cal structure, the IOP (21 A, 21 B) interfacing with a 
controller (30) at the next layer of the hierarchy, the 
method comprising the steps of> 

a) receiving events data from said field devices 
by the receiving IOP (21 A, 21 B) 

b) maintaining a first buffer with events data, 
said first buffer being a circular list, said events 
data being time stamped by the receiving IOP 
(21 A) with a relative time of receipt and a sync 
message ID, the relative time of receipt being 
a number of clock times since receipt of a time 
synchronizing message from the controlller, the 
time synchronizing message further including 
the sync message ID, said events data in the 
first buffer being the events data received since 
a previous transmission of said events data to 
the controller (30), 

c) maintaining a second buffer with events data, 
said second buffer being a circular list, said 
events data being time stamped by the receiv- 
ing IOP (21 B) with the relative time of receipt, 
said events data in the second buffer being 
events data received within a predetermined 
period such that a history of events is main- 
tained for the predetermined period; 

d) upon request from the controller (30), trans- 
mitting the events data and the corresponding 
relative time of receipt from the first buffer to 
the controller (30) whereby the controller (30) 
calculates a real time of occurrence of the 
events data, the real time of occurrence being 
time synchronized to the process control sys- 
tem (10); and 

e) when a failover is detected, upon a request 
for event recovery from any of the hierarchical 
layers, recreating the events data from the sec- 
ond buffer to transmit the recreated ev nts data 
to the controller (30) in accordance with a pr - 



defined protocol thereby avoiding the loss of 
any events data as a result of the failover. 

2. A method according to Claim 1 characterised in that 
5 the step of maintaining a second buffer comprises 
the steps: 

a) during a failover, freezing a purge operation 
of the events data in the second buffer even 

10 though the events data is older than the prede- 

termined period of time; 

b) continuing to store events data in the second 
buffer during the failover thereby not losing any 
events data; and 

15 c) upon completing processing events data in 

the second buffer, unfreezing the purge opera- 
tion of the second buffer to eliminate out of date 
events data. 

20 3. a process control system having a redundant mul- 
tilayer multimode hierarchical structure, the system 
programmed to operate a method as defined in 
Claim 1 or 2. 
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Patentanspruche 

1. Verfahren zur Aufrechterhaltung einer Folge von 
Ereignisfunkstionen wahrend einer Sicherheits- 
ubertragung von irgendwelchen redundanten Kno- 
ten in einem ProzeBsteuersystem (10) mit einer 
redundanten mehrschichtigen hierarchischen 
Struktur mehrerer Knoten, wobei jeder Knoten einer 
Schicht redundant ist, jede Folge von Ereignisein- 
gangen, die von Feldgeraten durch einen Emp- 
fangs-Ein/Ausgabe-Prozessor (IOP) (21 A,21 B) 
empfangen werden, eine digitale Eingangsfolge 
von Ereignissen (DISOE) IOP ist, der IOP die un- 
terste Schicht der hierarchischen Struktur ist und 
der IOP (21A.21B) mit einer Steuerung (30) der 
nachsten Schicht der Hierarchie eine Schnittstelle 
bildet und wobei das Verfahren die Schritte umfaGt: 

a) Empfangvon E reign isdaten von Feldgeraten 
durch den empfangenden IOP (21A.21B); 

b) Beibehalten von Ereignisdaten in einem er- 
sten Puffer, wobei der erste Puffer eine zirkula- 
re Liste ist, die Ereignisdaten durch den emp- 
fangenden IOP (21 A) mit einer relativen Emp- 
fangszeit und einer Synchronisations-Nach- 
richten-ID gestempelt sind, die relative Emp- 
fangszeit einer Anzahl von Taktzeiten seit dem 
Empfang einer Ze it-Synch ronisationsnachricht 
von der Steuerung ist, die Zeit-Synchronisati- 
onsnachricht ferner die Synch ron isations- 
Nachrichten-ID umfalM, wobei die Ereignisda- 
ten in dem ersten Puffer Ereignisdaten sind, die 
seit iner vorangegangenen Obertragung von 
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15 epo; 

Ereignisdaten zu der Steuerung (30) empfarv 
gen wurden; 

c) Beibehalten von Ereignisdaten in ein m 
zweiten Puffer, wobeiderzweite Puffer eine zir- 
kulare Liste ist, die Ereignisse durch den emp- 
fangenden IOP (21 B) mit einer relativen Emp- 
fangszeit gestempeft sind, die Ereignisdaten in 
dem zweiten Puffer Ereignisdaten sind, die in- 
nerhalb einer vorbestimmten Periode empfan- 
gen werden, so daB ein Verlauf von Ereignis- 
sen fur die vorbestimmte Periode aufrechter- 
halten wird; 

d) bei Aufforderung von der Steuerung (30), 
Ubertragung der Ereignisdaten und der ent- 
sprechenden relativen Empfangszeit von dem 
ersten Puffer zu der Steuerung (30), wobei die 
Steuerung (30) eine Echtzeit des Auftritts der 
Ereignisdaten berechnet und die Echtzeit des 
Auftritts mit dem ProzeBsteuersystem (10) zeit- 
synchronisiert ist; und 

e) wenn eine Sicherheitsubertragung festge- 
stellt wird bei einer Aufforderung nach einer Er- 
eignis-Regenerierung von irgendeiner der hier- 
archischen Schichten, Neubildung der Ereig- 
nisdaten aus dem zweiten Puffer, um die neu 
gebildeten Ereignisdaten zu der Steuerung 
(30) gemaB einem vorbestimmten Protokoll zu 
ubertragen, wodurch der Verlust irgendwelcher 
Ereignisdaten infolge der Sicherheitsubertra- 
gung vermieden wird. 

2. Verfahren nach Anspruch 1, dadurch gekenn- 
zeichnet, daB der Schritt der Beibehaltung eines 
zweiten Puffers die Schritte umfaBt: 

a) wahrend einer Sicherheitsubertragung das 
Einfrieren einer Ausspuloperation von Ereig- 
nisdaten in dem zweiten Puffer, obgleich die Er- 
eignisdaten alter als die vorbestimmte Zeitpe- 
riode sind; 

b) Fortfahren mit dem Speichern von Ereignis- 
daten in dem zweiten Puffer wahrend der Si- 
cherheitsubertragung, wodurch irgendwelche 
Ereignisdaten nicht verlorengehen; und 

c) bei Vervollstandigung der Verarbeitung von 
Ereignisdaten in dem zweiten Puffer, Freige- 
ben der Ausspuloperation des zweiten Puffers, 
um Ereignisdaten herauszueliminieren. 

3. ProzeBsteuersystem mit einer redundanten mehr- 
schichtigen hierarchischen Struktur mehrerer Kno- 
ten, wobei das System programmiert ist, um ein 
Verfahren auszufuhren, wie es in Patentanspruch 1 
Oder 2 definiert ist. 
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R vendi at ions 

1 . Precede destine a maintenir une sequence de fonc- 
tion d'evenements pendant une panne de n'importe 

s quel noeud redondant dans un systeme de com- 
mando de processus (10) ayant une structure hie- 
rarchique multinoeuds multicouches redondante, 
chaque noeud d'une couch e etant redondant, une 
sequence d'entrees d'evenements etant recue de 

10 dispositifs de terrain par un processeur d'entree/ 
sortie (IOP) de reception (21 A, 21 B) qui est un IOP 
de sequence d'evenements a entree numerique 
(DISOE), I'lOP etant la couche la plus basse de la 
structure hierarchique, I'lOP (21 A, 21 B) assurant 

'5 I'interface avec un circuit de commande (30) au ni- 
veau de la couche suivante de la hierarchie, le pro- 
cede comportant les etapes consistant a : 

a) recevoir des donnees d'evenements prove- 
20 nant desdits dispositifs de terrain avec I'lOP de 

reception (21 A, 21 B), 

b) maintenir un premier tampon avec des don- 
nees d'evenements, ledit premier tampon etant 
une liste circulaire, lesdites donnees d'evene- 

26 ments etant marquees temporellement par 

HOP de reception (21 A) avec un temps de re- 
ception relatif et un identificateur ID de messa- 
ge de synchronisation, le temps relatif de re- 
ception etant un nombre de temps d'horloge 

30 depuis la reception d'un message de synchro- 

nisation temporel provenant du circuit de com- 
mande, le message de synchronisation tempo- 
re! comprenant en outre I'lD de message de 
synchronisation, lesdites donnees d'6v6ne- 

3S ments dans le premier tampon etant les don- 

nees d'evenements recues depuis une trans- 
mission precedents desdites donnees d'eve- 
nements vers le circuit de commande (30), 

c) maintenir un deuxieme tampon avec des 
40 donnees d'6v6nements, ledit deuxieme tam- 
pon etant une liste circulaire, lesdites donnees 
d'6v6nements etant marquees temporellement 
par I'lOP de reception (21 B) avec le temps de 
reception relatif, lesdites donn6es d'ev6ne- 

<5 ments dans le deuxieme tampon etant des don- 

nees d'evenements recues pendant une perio- 
de predetermines de telle sorte qu'un histori- 
que d'evenements est maintenu pour la perio- 
de pr6determin6e; 

so d) lors d'une demande provenant du circuit de 

commande (30), emettre les donnees d'evene- 
ments et le temps de reception relatif corres- 
pondent provenant du premier tampon vers le 
circuit de commande (30) de sorte que le circuit 

ss de commande (30) calcule un temps r6el d'ap- 

parition des donnees d'evenements, le temps 
reel d'apparition etant synchronise dans le 
temps avec le system de commande de pro- 
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c ssus (10); et 

e) lorsqu'un panne est detectee, lors d'une d - 
mande d recuperation d'evenement prove- 
nant d Tune quelconqu des couches hierar- 
chiques, recr6er les donnSes d'eveYiements a s 
parti r du deuxieme tampon afin de transmettre 
les donn6es d'evdnements recedes au circuit 
de commande (30) en fonction d'un protocole 
prectefini en evitant ainsi la perte de donnSes 
d'ev6nements quelconques du fait de la panne. 10 

Proced6 selon la revendication 1 , caract6ris6 en ce 
que l'6tape de maintien d'un deuxieme tampon 
comporte les elapes consistant a : 

is 

a) pendant une panne, geler une operation de 
purge des donnSes d'6v6nements dans le 
deuxieme tampon m6me si les donn6es d'eve- 
nements sont plus anciennes que la periods de 
temps pr6d6termin6e; 20 

b) continuer a stocker des donn6es d'evene- 
ments dans le deuxieme tampon pendant la 
panne en ne perdant ainsi aucune donn6e 
d'6v6nement; et 

c) a la fin du traitement des donnees d'Svene- 25 
ments dans le deuxieme tampon, d6geler l'op6- 
ration de purge du deuxieme tampon afin d'eli- 
miner les donn6es d'evSnements depass^es. 

Systeme de commande de processus ayant une 30 
structure hierarchique multinoeuds multicouches 
redondante, le systeme 6tant programm6 afin de 
mettre en oeuvre un procedd selon la revendication 
1 ou 2. 

35 
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